Sound Mosaics

This thesis presents the design of a Graphical User Interface (GUI) for computer-based sound synthesis to support users in the externalisation of their musical ideas when interacting with the system in order to create and manipulate sound. The approach taken consisted of three research stages. The first stage was the formulation of a novel visualisation framework to display perceptual dimensions of sound in visual terms. This framework was based on the findings of existing related studies and a series of empirical investigations of the associations between auditory and visual percepts that we performed for the first time in the area of computer-based sound synthesis. The results of our empirical investigations suggested associations between the colour dimensions of brightness and saturation with the auditory dimensions of pitch and loudness respectively, as well as associations between the multidimensional percepts of visual texture and timbre. The second stage of the research involved the design and implementation of Sound Mosaics, a prototype GUI for sound synthesis based on direct manipulation of visual representations that make use of the visualisation framework developed in the first stage. We followed an iterative design approach that involved the design and evaluation of an initial Sound Mosaics prototype. The insights gained during this first iteration assisted us in revising various aspects of the original design and visualisation framework that led to a revised implementation of Sound Mosaics. The final stage of this research involved an evaluation study of the revised Sound Mosaics prototype that comprised two controlled experiments. First, a comparison experiment with the widely used frequency-domain representations of sound indicated that visual representations created with Sound Mosaics were more comprehensible and intuitive. Comprehensibility was measured as the level of accuracy in a series of soundimage association tasks, while intuitiveness was related to subjects' response times and perceived levels of confidence. Second, we conducted a formative evaluation of Sound Mosaics, in which it was exposed to a number of users with and without musical background. Three usability factors were measured: effectiveness, efficiency, and subjective satisfaction. Sound Mosaics was demonstrated to perform satisfactorily in all three factors for music subjects, although non-music subjects yielded less satisfactory results that can be primarily attributed to the subjects' unfamiliarity with the task of sound synthesis. Overall, our research has set the necessary groundwork for empirically derived and validated associations between auditory and visual dimensions that can be used in the design of cognitively useful GUIs for computer-based sound synthesis and related areas. Acknowledgements First and foremost, I would like to thank my supervisors, Prof. Ann Blandford and Dr. Matt Smith for providing all the time and support needed to complete my studies. It has been an incredible experience to work with you, learn from you, and be inspired by you. Thanks are also due to the research team at Middlesex University who gave me the opportunity and the resources to do this research. I would also like to thank all the students and staff at the Sonic Arts Dept. (Middlesex University) as well as my fellow research students who participated in the experiments described in this thesis. Particular thanks go to Dr. John Dack, Andrew Deakin, Martin Robinson, and Tony Gibbs for making the Sonic Arts Dept. feel like a second home. Special thanks to Thomas Tan and Penny Duqenoy for their support and friendship. Other people who I would like to thank for making my time at Middlesex University such a pleasant experience include: Casper Nielsen, Doreen Ng, Kok Fong Tan, Sardia Alhassan, and Dr. Serengul Smith. My interest in sound synthesis and novel approaches to music composition would not have been the same without some inspiring discussions I had with my best friend and artist Nicolas Teloglou. My heartfelt thanks to Nasia for her love and patience all these years. Finally, I cannot find words to express my love and gratitude to my family for their unconditional love and support throughout my life. This is dedicated to you.